Skip to content

perf: optimize levenshtein#23543

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/levenshtein-datafusion-20260713-191815
Open

perf: optimize levenshtein#23543
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:auto-opt/levenshtein-datafusion-20260713-191815

Conversation

@andygrove

@andygrove andygrove commented Jul 14, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

Rationale for this change

Optimize existing expression.

What changes are included in this PR?

Added Myers' bit-parallel Levenshtein fast path for ASCII inputs whose shorter side fits in a 64-bit word, replacing the per-cell character-wise DP with a few bitwise ops per text character.

Are these changes tested?

Existing tests + new unit tests.

Benchmark (criterion):

  • levenshtein_string [size=1024, str_len=32]: 83.81% faster (base 662708ns -> cand 107290ns)
  • levenshtein_string [size=1024, str_len=8]: 54.904% faster (base 58165ns -> cand 26230ns)
  • levenshtein_string [size=4096, str_len=32]: 83.79% faster (base 2640918ns -> cand 428103ns)
  • levenshtein_string [size=4096, str_len=8]: 54.274% faster (base 227761ns -> cand 104146ns)

Full criterion output:

levenshtein size=1024/levenshtein_string [size=1024, str_len=8]
                        time:   [26.188 µs 26.230 µs 26.298 µs]
                        change: [−55.035% −54.904% −54.755%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe
levenshtein size=1024/levenshtein_string [size=1024, str_len=32]
                        time:   [107.24 µs 107.29 µs 107.35 µs]
                        change: [−83.837% −83.810% −83.768%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) high mild

levenshtein size=4096/levenshtein_string [size=4096, str_len=8]
                        time:   [104.07 µs 104.15 µs 104.24 µs]
                        change: [−54.330% −54.274% −54.218%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
levenshtein size=4096/levenshtein_string [size=4096, str_len=32]
                        time:   [427.90 µs 428.10 µs 428.33 µs]
                        change: [−83.799% −83.790% −83.780%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) high mild

Are there any user-facing changes?

No

@github-actions github-actions Bot added the functions Changes to functions implementation label Jul 14, 2026
@andygrove andygrove changed the title perf: optimize levenshtein in datafusion-functions perf: optimize levenshtein Jul 14, 2026
@andygrove andygrove marked this pull request as ready for review July 14, 2026 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant